Grouping of Color and Texture Features for Automated Image Annotation

We present a system for automated image annotation which is capable of detecting concepts such as sky, water and man-made structure in color images. The processing of each image consists of a feature extraction stage and a grouping stage. The ensuing concept-recognition is accomplished using a decision tree. The pixel-level features consist of basic color and texture information. The color information consists of hue, saturation and value (intensity) data and the texture information is obtained using the windowed-image second moment matrix. The pixel-level features are quantized into one of a dozen or so bins based on an empirically determined perceptual partitioning of color/texture space. The set of binary images associated with this quantization step are each grouped in parallel according to three diierent strategies. The three grouping strategies seek to form regions according to (1) solid contiguity, (2) similarity in local orientation and (3) similarity in diiuseness. As an example, one of the above mentioned binary images contains a 1 at each point where the original image contains a pixel with a bluish hue. Should the input image contain clear blue sky above the horizon, the rst grouping strategy would produce a large connected region in the binary image representing the pixels with a light-blue color. The second grouping strategy would abort due to lack of orientation strength and the third strategy would fail since the sky-blob is not diiuse. Each blob is represented by a feature vector containing its area, coordinates in the image, eccentricity , principle orientation, mean saturation and mean intensity, as well as by the color/texture bin and grouping strategy which gave rise to it. These feature vectors are the input to the decision tree classiier. The decision tree attempts to assign a label to each blob according to these characteristics.